Skip to content

Check every count the docs state against the tree (tools/claims.py) - #88

Merged
shuv-amp merged 5 commits into
mainfrom
stack/01-claim-checker
Oct 17, 2026
Merged

shuv-amp merged 5 commits into
mainfrom
stack/01-claim-checker

Conversation

@shuv-amp

@shuv-amp shuv-amp commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Opened by the Bulkhead autopilot agents: one reviewed unit of work. Every commit was reviewed adversarially by the other agent, with mutants, and passes tools/claims.py on its own.
The agents never merge. The owner reviews and merges.

Unit 1 of 4 (agent B). tools/claims.py, run by scripts/check.sh: every count README, docs/, ci.yml and the status page state is computed from the tree, and drift fails CI.

Why

On main it finds 15 stale counts: Kani 15→17, program tests 123→133, fuzz targets 8→9, Floor tests 41→45 and more. All are fixed here, from the tree.

Commits

  1. The checker and the fixed counts. Each rule was shown red on a planted stale count, then green (agent A re-ran this independently).

  2. The post-submission findings range (96–104) as a checked rule.

  3. A kept statement exempts only the numbers inside it (owner's review): before, one dated "16/16 verified" hid a stale harness count in the same row. With the old rule and the new self-test case: "self-test FAILED … 0 errors, expected 1". Fixed: self-test OK, 40 claim cases.

  4. Drop the findings-range lookbehind (owner's second review): after commit 3 it was a workaround for a problem that no longer exists, and its comment was false. The plain rule still catches a planted "96–103 recorded".

Limits

  • Counts come from reading source, not from test runs.
  • A count in a phrasing no rule names is not checked (for example "33 commands", "1,500 limiter sessions").

No program bytes change.

🤖 Generated with Claude Code

shuv-amp and others added 3 commits October 16, 2026 00:08
What was wrong: the public counts had drifted from the code, and nothing
caught it. At 54decc4 the README badge, results table, FAQ and SECURITY.md
said 15 Kani harnesses (the core has 17, 09 says 17/17); README said 98
findings (09 §4 has 103 rows, 1..103 without a gap); 123 program tests
(Floor 43, Vault 80) where floor.rs and vault.rs hold 45 and 86; 8 fuzz
targets (fuzz/Cargo.toml has 9 [[bin]]s); 10 Floor core tests (12); the
threat model's I15 cited 41 Floor tests; README and 09 said the
differential runs 31 commands.

Root cause: every count was typed by hand in several places, and no check
compared any of them with the code.

Fix: tools/claims.py (stdlib Python, like floor_check.py) computes each
count from the files themselves: Kani harnesses by name, from their
attributes; test functions per suite file, without #[ignore]d ones and
reading past comments, literals and nested attribute brackets; the JS
tests outside comments and strings, less skipped/todo ones; golden
outputs and regression blocks as golden.py --check compares them; the
fuzz [[bin]]s; the rows of 09 §4. It then matches 37 phrasings of those
counts in every Markdown file in git's index, the CI workflow and the
status page, case-insensitively and across line breaks, and fails on any
mismatch. A passed/total result must be all passed, since a run with
failures is a red CI, not a state the docs describe. More checks:
- 09 §5.1's table must list exactly the harnesses in the source. The
  README's "14 for every value" and 09's "3 at small width" are derived
  from it.
- Every file in the two LiteSVM/core test directories must be mapped to
  a count or listed with the reason it isn't.
- A rule that matches no current statement fails, so rewording a
  sentence can't leave its rule checking nothing.
- KEPT holds statements of a past state, each with its reason: the
  decision log's 14 harnesses and the submission's 95 findings. Each must
  match exactly once and overlap a rule match.
scripts/check.sh runs the self-test and the check right after fmt, so
CI's always-on `check` job fails fast on a stale count.

Docs corrected to the tree, each from a run in this branch:
- Kani 17/17. scripts/prove.sh on 15 Oct: "Complete - 17 successfully
  verified harnesses, 0 failures, 17 total" (4:43 wall). The README cites
  09 §5.1's 14 Oct run (227 s), so one run is quoted in both places.
- 131 program tests. scripts/check-programs.sh: floor 45 passed,
  vault 86 passed; curve_vectors and e2e_world 0 passed, 1 ignored each.
  sha256 floor 508e5a99…, vault b07655d9…, harbor 543be60b…
- Core suites from cargo test: properties 26, exhaustive 6, floor 12,
  units 25, adversarial 7, cascade 2. node --test: 176. All equal to
  the static counts.
- 103 findings, with the post-submission sentence restated: 96–103 are
  fixed in this tree but not yet on devnet, except 100 and 102, which are
  stated.
- 9 fuzz targets. The 6 Oct 10-minute run had 8, and the guardian-set
  target has no recorded 10-minute run or coverage measurement, so the
  README and 09 now say that.
- 33 differential commands, the distinct tags in check.sh's seed-11 run.
- docs/12: 96–103 recorded on 13 and 14 Oct.
docs/08's "+ 42 more Floor tests" stays as written: its status table is
dated 10 Oct, when floor.rs had 43.

Proof the check can fail, each mutant red and then reverted to green:
- Docs: FAQ back to 15; an 18th harness not in 09 §5.1; a Vault test
  added (132 ≠ 131); a 104th finding row; a 10th fuzz bin; a 177th JS
  test; README back to 98 findings; a 09 §4 row deleted ("row 50 is
  numbered 51"); 09's "17 total" set to 16; a regression block dropped;
  04's "36 fixtures" set to 35; an unmapped tests/zz/main.rs.
- Checker logic, caught by the self-test: ignore not excluded; no number
  lookbehind; spaces matched literally; never comparing; stale or dead
  KEPT entries passing; kept spans not skipped or only matched at their
  start; no word boundary on KEPT; liveness not tracked; nesting depth
  ignored; attribute run not broken by code; raw strings, multi-char
  escapes or char literals not lexed; JS strings not lexed or dropped;
  skip:false read as skipped; cfg_attr/proof_for_contract harnesses
  missed; comments counted.
- An unanchored line-comment pattern hung on 40 `//` (killed at 120 s).
  The test-count scanner now strips comments first.
One equivalent mutant: resetting attributes after a non-fn item, since
the item's own text resets the run anyway.

Review: seven code-review passes (high). Every valid finding was fixed.
Declined, with reasons:
- "136 program tests": curve_vectors and e2e_world are #[ignore]d (run
  log above).
- Checking only totals in passed/total: it would let a stale "15/17"
  stand. A results file written by the run itself is the real fix, now
  in the backlog.
- "N findings" matching subsets: a false positive fails loudly, and
  KEPT takes a subset with its reason.
- Narrowing ci-changes.sh's tools/* selector: that runs fewer jobs, so it
  is its own reviewed change.
Known limits, stated in the code: counts come from source text, so a
cfg-gated test or one registered in a JS loop needs a run to confirm.

Sibling search: git grep for every count phrasing above across *.md,
ci.yml and cloud/status. ci.yml's "core's 17 harnesses" and 09 §5.1's
"Complete … 17 total" are now ruled too.

No program source changed (git diff on crates/ and programs/ is empty),
so the sBPF builds are unchanged. scripts/check.sh: ALL CHECKS PASSED.

Rebased onto main 1045cd9 (#85, #86) for its own PR: README and docs/12
keep main's devnet and findings rows (devnet runs PR #85; 104
findings); counts are main's tree: programs 133 (Floor 45, Vault 88).
claims.py --self-test OK, claims.py OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README and 12 name the findings since the submission as a range,
"96–104". claims.py only knew the phrasing "N findings", so when a
finding is added the range goes stale and the check still passes. It
did once already: main's 104 left "96–103" in both (fixed on main by
#86).

- claims.py: one rule per sentence ties the range end to the 09 §4 row
  count: README "96–N were found after the submission", 12 "…at
  submission; 96–N recorded". One rule each, so rewording either
  sentence fails the "matches no current statement" guard. Locked to
  the wording because README's "96–99, 101 and 103 are fixed" (a
  sub-range) must not match. Either dash. The README rule is not read
  after a digit (WHOLE, now one guard shared by N, that rule and the
  KEPT matcher). The 12 rule's context is a lookbehind, which also does
  that job: a match overlapping the kept "95 findings at submission"
  would be skipped, and "96–N recorded" elsewhere must not match. Any
  whitespace after the semicolon. Self-test cases for each rule and
  branch, including a sub-range before the range, "96–N recorded"
  outside 12's sentence, the kept entry beside the range, and a
  reflowed line.
- 05 §10's caption said the figures come from a build of "this tree".
  They come from #35's (25b5a87, 8 Oct; `git log -S` on the figures),
  and 12 commits dated after 8 Oct change Vault or core source. The
  caption now says so, and names the two later figures and their
  sources: a first deposit at a low bump (finding 75) and the full-ledger
  row (its own test, 14 Oct). S8's "< 1.4M" is the transaction limit,
  as its row says.

The wording of README and 12 is main's (#86). An earlier version of this
commit (55f4a7b, never integrated) reworded those sentences itself; #86
has since rewritten them, so only the rules move across, adapted.

Evidence (tasks/sync-main/evidence.sh, its log final-evidence.log):
claims.py OK, findings 104. Plants, each exit 1 and restored: README
96–103 and 96–105 ("states 103/105, the tree has findings = 104"); 12
96–103, also reflowed with two spaces; README "were found" → "came"
and 12 "recorded" → "logged" ("rule … matches no current statement"),
the latter also with "96–104 recorded" planted in README. "96–103
recorded" planted in README alone: exit 0 (not read). Self-test
mutants, each red: delete either rule, drop the README rule's digit
guard, narrow either to an en dash, loosen either to a bare "96–N",
drop the 12 rule's context, make it a plain prefix instead of a
lookbehind, allow exactly one space. Restored: "self-test OK: 39 claim
cases, 22 count cases, 39 rules".

Not checked: the fixed/stated split in the same sentences ("96–99, 101
and 103 are fixed…; 100, 102 and 104 are stated") and 12's dates ("13–14
Oct"). When finding 105 lands, check.sh fails until both ranges say
105, but nothing makes the split or the date follow. Also left (code
review, pre-existing, backlog 1c): FAQ's "~10.3k" Floor stop vs 06 §7's
10,461, 06 §7's "this tree", and 09 §2's "(10 Oct 2026)" heading above
today's counts, and 05 §10's Target column (≤ 25k deposit, ≤ 35k G2),
where the test allows 12k more for Harbor's CPI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check() skipped a whole rule match when it overlapped a KEPT statement
at all, so a current count matched by the same rule (the harness count
in a Kani table row whose dated "16/16 verified" is kept) went
unchecked. The self-test asserted that: "16 harnesses ... **16/16
verified**" with 16/16 kept gave 0 errors against kani = 17.

Now each captured number is exempt only when it lies inside a kept
statement; the rest of the match is checked, and a match counts as live
whenever any of its numbers is checked.

Self-test: that case now expects 1 error, and a new case (17 harnesses,
the same kept 16/16) expects 0. With the old check() and the new cases:
"self-test FAILED: ... 0 errors, expected 1". With the fix: self-test
OK (40 claim cases, 22 count cases, 39 rules); claims OK on the tree,
so no stale count was hiding behind a kept statement.

Found by the owner's code review of PR #87.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shuv-amp and others added 2 commits October 16, 2026 01:21
docs/12's range rule used a lookbehind, with a comment that a match
overlapping the kept "95 findings at submission" would be skipped
whole. Since the previous commit only a kept statement's own numbers
are exempt, so that comment was false and the workaround unneeded.
The rule is now the plain "submission; 96–N recorded".

Evidence: self-test OK (40 claim cases); claims OK on the tree; a
planted "96–103 recorded" in docs/12 is caught: "states 103 (the
tree has findings = 104)".

Found by the owner's second code review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README conflict is resolved to main's new layout with this branch's
corrected counts: Kani 17/17 (14 Oct, 14 at full width), 133/133 program
tests, 9 fuzz targets with the note on the unmeasured one, 33 differential
commands, and tools/claims.py in the repository layout. Two rules now accept
the new table rows, whose label is followed by a caption.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shuv-amp
shuv-amp enabled auto-merge (squash) October 17, 2026 17:48
@shuv-amp
shuv-amp merged commit acf91a1 into main Oct 17, 2026
11 checks passed
@shuv-amp
shuv-amp deleted the stack/01-claim-checker branch October 17, 2026 17:53
shuv-amp added a commit that referenced this pull request Oct 17, 2026
Conflicts resolved: this branch's counts in docs/09 (136 program tests,
Vault 91), main's tools/claims.py, and main's README with 136/136.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant